Skip to content

llm: pass --model to every backend, replace gemini with agy, report real spend - #133

Merged
MarkusNeusinger merged 2 commits into
mainfrom
claude/llm-backend-model-and-usage-reporting
Jul 28, 2026
Merged

llm: pass --model to every backend, replace gemini with agy, report real spend#133
MarkusNeusinger merged 2 commits into
mainfrom
claude/llm-backend-model-and-usage-reporting

Conversation

@MarkusNeusinger

@MarkusNeusinger MarkusNeusinger commented Jul 28, 2026

Copy link
Copy Markdown
Owner

What

Closes three related gaps in how citadel talks to and accounts for its agent CLIs.

1. CITADEL_INGEST_MODEL reaches every backend. It used to be a claude-only knob, hard-defaulted
to sonnet, with doctor actively warning that it was inert on the others. All three CLIs accept
--model, so it is now passed to all three — and defaults to unset, so an unconfigured workspace
gets each CLI's own default instead of a value citadel invented.

2. gemini -> agy. The Gemini CLI no longer exists; it is now Google's Antigravity CLI (agy),
with different flags (--dangerously-skip-permissions, repeatable --add-dir, no --session-summary,
so the statfile machinery is gone). CITADEL_LLM_CLI=gemini fails loudly with a migration message.

3. The stamped model is the model that actually ran. It was previously assembled from .env — a
guess that could name a model no session ever used (notably with a local Ollama-backed claude). Each
backend's own session envelope is now parsed for the effective model, and reported wins over the
config label, which stays only as the fallback. copilot's spend is read as AI credits
(totalNanoAiu) and priced at GitHub's published $0.01/credit so a mixed corpus still has one
comparable total; it reports no prompt tokens, so tokens_in honestly stays unset.

Surfaced where it is consumed: sources/index.md grows Model/Cost/Tokens/Checked columns, the
viewer shows the same provenance in the source reader and hover popover, and status reports AI
credits alongside USD.

Ingest console rewritten on rich (requested in-thread). A live region with one spinner row per
in-flight source — which repairs --jobs N, where the animation used to be switched off entirely
because a single rewritten line could name only one source — plus a permanent verdict line per
source carrying its cost, tokens and model:

[1/1] OK  raw/console-demo.md  45.1s  1 created  $0.4431  2k out  44.3073 AIC  claude-opus-5

ASCII output is preserved (the spinner is pinned to rich's line frames); off-TTY and --quiet
degrade to the previous plain lines. rich is added as a runtime dep (floor >=13, pure-Python,
two small pure-Python transitive deps).

Console-flooding fix. config.display_key only shortened a key that matched a configured root
as a string, so a Windows drive mapped to a share (T:\proj\raw resolving to
//fileserver.long.example/proj/raw/...) printed the full UNC path on every line. It now falls back
to the last path segment matching a root's folder name, then to a marked tail clip. Verdict lines
also clip the path from the left rather than dropping the spend off the right when the terminal is
narrow.

Testing

  • uv run pytest -q — 1277 passed, 1 skipped. Two failures are pre-existing and environmental, not
    regressions: test_ingest_staging.py::test_agent_edits_staging_sibling_not_live picks up the local
    workspace .env (CITADEL_WIKI_DIR), and test_wikigit.py::test_push_to_a_configured_remote
    depends on local git config. Both were confirmed by reverting to main.
  • uv run ruff check . / uv run ruff format --check . — clean
  • CITADEL_WORKSPACE=corpora/beverages uv run python -m citadel lint — OK
  • Live, in WSL with all three CLIs installed: one real citadel ingest per backend
    (claude / copilot / agy) in a throwaway workspace, verifying --model is honored, the real
    model ID is stamped, usage arrives, and the viewer/sources catalog render it. The rich console was
    additionally verified in a real run and under a PTY at 80 and 100 columns.
  • verify-corpus was skipped at the owner's request — the three live ingests above exercise the
    ingest/llm path end to end.

🤖 Generated with GitHub Copilot CLI

@MarkusNeusinger
llm: pass --model to every backend, replace gemini with agy, report r…
962f484 
…eal spend

Three related gaps in how citadel talks to and accounts for its agent CLIs:

- CITADEL_INGEST_MODEL was a claude-only knob hard-defaulted to sonnet and inert
  elsewhere. All three backends accept --model, so it is passed to all three and
  now defaults to unset (the CLI's own default).
- The gemini CLI no longer exists; it is replaced by agy (Google's Antigravity
  CLI), with a loud migration error on the retired name.
- The model stamped per source was a guess assembled from .env. Each backend's
  own session envelope is now parsed for the model that actually served, and
  that is what the manifest, status, the sources catalog and the viewer record.
  copilot's spend is captured in AI credits (totalNanoAiu) and priced at
  GitHub's published $0.01/credit, so a mixed corpus has one comparable total.

Model and usage are surfaced where they are consumed: per-source columns in
sources/index.md, the viewer's source reader and popover, and the ingest console
— which is rewritten on rich. It now renders a live region with one spinner row
per in-flight source (repairing --jobs N, which previously turned the animation
off because a single rewritten line could name only one source) and prints a
permanent verdict line per source carrying its cost, tokens and model.

Also fixes a console-flooding bug: display_key only shortened a key that matched
a configured root as a string, so a Windows drive mapped to a share printed the
full UNC path on every line. It now falls back to the root's folder name and
then to a marked tail clip, and verdict lines clip the path rather than the
spend when the terminal is narrow.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 28, 2026 18:11
Copilot AI reviewed Jul 28, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR modernizes citadel’s agent-CLI integration and provenance accounting: it passes --model consistently across backends, migrates the retired gemini backend to Google’s agy, and surfaces backend-reported model/usage (including Copilot AI credits) through the manifest, status output, sources catalog, and offline viewer.

Changes:

  • Standardize model selection by passing --model to claude, copilot, and agy, with CITADEL_INGEST_MODEL defaulting to unset (CLI default).
  • Replace the removed gemini CLI integration with agy and update docs/tests accordingly.
  • Expand provenance reporting (model that actually ran, cost/tokens/AIC, checked date) across manifest consumers and rewrite the ingest console UI using rich.

Reviewed changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
uv.lock Adds rich (+ transitive deps) to the locked environment.
pyproject.toml Adds rich>=13 to runtime dependencies with rationale.
citadel/progress.py Replaces the hand-rolled progress spinner with a rich live region and richer per-source verdict lines.
citadel/config.py Changes model configuration defaults and improves display_key shortening (mapped-drive/UNC handling + clipping).
citadel/manifest.py Extends usage stamp handling (adds aic, defensive rounding/filtering) and clarifies model stamping semantics.
citadel/ingest.py Stamps model provenance from backend-reported model, threads usage/model into progress events, and updates accounting comments.
citadel/status.py Adds AI-credits totals and rendering of AIC alongside USD in citadel status.
citadel/catalogs.py Expands sources/index.md catalog with Cost/Tokens/Checked columns sourced from manifest stamps.
citadel/viewer/init.py Includes manifest usage/checked provenance in the viewer bundle for sources.
citadel/viewer/app.js Renders source provenance (model/cost/AIC/tokens/checked) in reader + hover popover.
citadel/viewer/app.css Styles new provenance fragments in the source reader.
citadel/doctor.py Updates doctor checks to treat gemini as retired and to reflect new model-selection behavior.
citadel/cli.py Updates --verbose help text to reflect transcript behavior for agy vs gemini.
citadel/templates/env.example Updates workspace .env template for agy and backend-agnostic CITADEL_INGEST_MODEL.
citadel/resume.py Updates backend references in resume design commentary (copilot/agy).
tests/test_usage_accounting.py Reworks accounting tests for Copilot JSONL + AIC, Claude model extraction, and agy stream-json usage parsing.
tests/test_progress.py Updates progress/display-key tests for rich rendering and new clipping/formatting behavior.
tests/test_ingest_progress.py Updates progress event contract tests to include per-source usage/model.
tests/test_ingest_parallel.py Updates parallel ingest progress expectations (live region preserved under --jobs).
tests/test_manifest.py Updates model label tests (backend-agnostic label + model_label_for).
tests/test_llm.py Updates invocation tests for agy flags and --model propagation; asserts gemini is refused.
tests/test_netdrive.py Updates external-dir grant tests from gemini to agy.
tests/test_viewer.py Adds viewer bundle provenance tests for usage/checked stamping behavior.
tests/test_ingest_provenance.py Updates sources catalog header expectations for new columns.
tests/test_doctor.py Updates doctor expectations and adds retired-backend failure test.
README.md Updates documented supported CLIs and setup instructions (agy instead of gemini).
SECURITY.md Updates threat model text for supported CLIs (agy).
docs/configuration.md Updates configuration reference for new model semantics, retired backend, and local-model guidance.
docs/troubleshooting.md Updates troubleshooting references (AGY_CLI_PATH).
docs/recipes.md Updates cron/CLI path override docs (AGY_CLI_PATH).
docs/maintenance.md Updates maintenance docs for new accounting/model provenance sources.
CHANGELOG.md Documents breaking backend migration + provenance/accounting/console changes.
CLAUDE.md Updates contributor guidance to reflect new dependencies, backends, and accounting semantics.
.github/copilot-instructions.md Regenerated Copilot instructions to match updated CLAUDE.md guidance.
.github/ISSUE_TEMPLATE/bug_report.yml Updates issue template backend options and description (agy).
.claude/skills/bench-model/SKILL.md Updates bench-model skill docs to reflect agy backend option.

Comment thread citadel/catalogs.py
Comment thread citadel/status.py
Comment thread citadel/viewer/app.js
@MarkusNeusinger
accounting: show AI credits even when dollars are unknown
61be84b 
cost_usd and aic are stamped independently, so a session can report credits
without a dollar figure. The sources catalog, the status table and the viewer
all keyed their spend rendering on cost_usd alone, so such a source rendered
as "—" while still contributing to the corpus credit total.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 28, 2026 18:18
Copilot AI reviewed Jul 28, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 36 out of 37 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

citadel/doctor.py:18

  • The module-level description says a retired backend name ("gemini") fails the ingest model check, but the actual failure happens in check_agent_cli() (which calls llm.resolve_cli_name). This mismatch can mislead users reading citadel doctor output/docs. Either move the retirement note under agent CLI or reword this bullet to match current behavior.
- **ingest model** — which model ingest will ASK for. Every backend (claude/copilot/agy) is passed
  ``--model``, so an unset ``CITADEL_INGEST_MODEL`` simply means "the CLI's own default"; a
  RETIRED backend name (``gemini``, now Antigravity's ``agy``) FAILs here with its migration hint.

@MarkusNeusinger
MarkusNeusinger merged commit b7f65d1 into main Jul 28, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MarkusNeusinger